Skip to content

test(ui): stop the comet retry loop spinning through the UI tests (CI ui_tests hang) - #684

Merged
brickbots merged 2 commits into
mainfrom
fix/ui-tests-wait-for-catalog-loader
Oct 29, 2026
Merged

brickbots merged 2 commits into
mainfrom
fix/ui-tests-wait-for-catalog-loader

Conversation

@brickbots

@brickbots brickbots commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

ui_tests has hung intermittently on CI since 2026-10-24, on main as well as on PRs. A CI thread dump found the cause: the UI test harness's own time.sleep stub turns the comet catalog's retry loop into a busy loop. Test-only fix; no runtime code changes.

Evidence

run commit result
main, after #677 0e0b2b1 cancelled at the 20-min job limit
main, after #680 c355928 cancelled at the 20-min job limit
#681 (twice) 57b231f cancelled at the 20-min job limit
#683 (twice) fca573f cancelled at the 20-min job limit
passing runs since 10-24 ui_tests alone took 150–400 s; locally ~80 s
  • In every cancelled run, lint, format, type_hints, smoke_tests and unit_tests all passed.
  • ui_tests then stalled at a different point each time.

Cause

This PR's first commit made a hanging test dump every thread's stack (faulthandler_timeout), and the next CI run hung and dumped them:

  • Main thread: still working, just crawling, inside a UILog.update draw.
  • Another thread: in comet_catalog.py retry_task → _download_once → the mock's _increment_mock_call.

CometCatalog._start_background_retry loops "time.sleep(60), download" until the comet file exists. The UI harness:

  • stubs comet_data_download to fail, so the file never appears;
  • no-ops time.sleep for the whole session (_fast_sleep).

So the loop never slept and never ended. It spun for the whole run, calling the mock (which records every call) and logging on every pass, and starved the tests until CI ran past the 20-minute limit.

A developer machine that already has a comet file in ~/PiFinder_data never starts the loop. That's why it never reproduced locally. With an empty HOME, a four-test slice logged 5,012 retries before the fix, and 0 after.

Both pieces date from #438 (May), so the loop has spun in every CI run since then. It only recently crossed the time limit, as the suite grew.

The change

  • _no_comet_download also stubs CometCatalog._start_background_retry. With no network, there's nothing to retry. This is the fix.
  • The catalogs fixture waits for the background CatalogLoader before yielding (first commit). It isn't the cause, but on CI (no catalog cache) the loader otherwise runs flat out beside the first tests, again without its per-batch yield, and rewrites the live catalogs when it finishes. Now every test sees fully loaded catalogs. Locally this adds about 6 s, once.
  • nox -s ui_tests sets -o faulthandler_timeout=180, so any future hang dumps every thread's stack to the log instead of the job dying silently. This is what found this one.

Testing

🤖 Generated with Claude Code

brickbots and others added 2 commits October 28, 2026 21:23
ui_tests has hung intermittently on CI since 2026-10-24: four of the last
six nox runs (main at 0e0b2b1 and c355928, #681, #683) stalled in
tests/test_ui_modules.py until the job's 20-minute limit, at different
points in the run, while every other session passed. Locally the suite
passes in about 80 s, warm or cold cache; CI's passing runs took 150-400 s.

The session-scoped catalogs fixture yielded while CatalogBuilder's
background CatalogLoader thread was still loading the deferred objects
(CI never has a catalog cache). The autouse _fast_sleep fixture no-ops
time.sleep, which removes the loader's per-batch yield, so it ran flat out
beside every test; and on completion its callback added objects to the
live catalogs, re-filtered them and pickled them to the cache, from that
thread, while tests were reading the same catalogs.

The fixture now joins the loader before yielding, so the tests run on
fully loaded catalogs with no thread mutating them. ui_tests also sets
pytest's faulthandler_timeout, so a test still running after 180 s dumps
every thread's stack to the log instead of the job dying silently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… tests

The CI thread dump from the previous commit's faulthandler_timeout found
the hang. A failed comet download starts CometCatalog's background retry,
which loops "sleep 60 s, download" until the comet file exists. In the UI
harness the download is stubbed to fail and _fast_sleep no-ops time.sleep,
so that loop never slept and never ended: it spun for the whole session,
calling and recording the mock and logging on every pass, until CI's tests
crawled past the job's 20-minute limit. A machine that already has a comet
file in ~/PiFinder_data never starts the loop, which is why it didn't
reproduce locally. Run with an empty home, a four-test slice logged 5,012
retries; with the retry stubbed out, none.

_no_comet_download now stubs _start_background_retry too: with no network
there is nothing to retry. The loader join from the previous commit stays,
reworded: it isn't the cause, but it keeps the loader from competing with
the first tests and every test sees fully loaded catalogs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@brickbots brickbots changed the title test(ui): wait for the background catalog loader before the UI tests run test(ui): stop the comet retry loop spinning through the UI tests (CI ui_tests hang) Oct 29, 2026
@brickbots
brickbots merged commit 317377c into main Oct 29, 2026
4 checks passed
Zhuohang-Shen pushed a commit to Zhuohang-Shen/PiFinder that referenced this pull request Nov 15, 2026
Brings the merge of brickbots main into deepchart into
observable-asteroids.

Conflicts and their resolution:
- ui/object_list.py: the imports of both sides (re for the names view
  of upstream, time for the asteroid branch).
- tests/test_ui_modules.py: the comet fixture keeps the
  check_if_comet_download_needed stub of the asteroid branch and adds
  the retry stub of upstream (brickbots#684).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant